Unsupervised Learning Project¶
Data Description:¶
The data contains features extracted from the silhouette of vehicles in different angles. Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars.
Domain:¶
Object recognition
Context:¶
The purpose is to classify a given silhouette as one of three types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.
Attribute Information:¶
All the features are geometric features extracted from the silhouette. All are numeric in nature.
Learning Outcomes:¶
Exploratory Data Analysis Reduce number dimensions in the dataset with minimal information loss Train a model using Principle Components
Objective:¶
Apply dimensionality reduction technique – PCA and train a model using principle components instead of training the model using just the raw data.
Steps and tasks:¶
- Data pre-processing – Perform all the necessary preprocessing on the data ready to be fed to an Unsupervised algorithm (10 marks)
- Understanding the attributes - Find relationship between different attributes (Independent variables) and choose carefully which all attributes have to be a part of the analysis and why (10 points)
- Split the data into train and test (Suggestion: specify “random state” if you are using train_test_split from Sklearn) (5 marks)
- Train a Support vector machine using the train set and get the accuracy on the test set (10 marks)
- Perform K-fold cross validation and get the cross validation score of the model (10 marks)
- Use PCA from Scikit learn, extract Principal Components that capture about 95% of the variance in the data – (10 points)
- Repeat steps 3,4 and 5 but this time, use Principal Components instead of the original data. And the accuracy score should be on the same rows of test data that were used earlier. (hint: set the same random state) (10 marks)
- Compare the accuracy scores and cross validation scores of Support vector machines – one trained using raw data and the other using Principal Components, and mention your findings (5 points)
In [1]:
#Import all the necessary modules
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
from scipy import stats
from sklearn import tree
from sklearn import preprocessing
from sklearn.preprocessing import LabelEncoder
from warnings import simplefilter # import warnings filter
simplefilter(action='ignore', category=FutureWarning) # ignore all future warnings
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
In [2]:
# Load dataset
df= pd.read_csv("vehicle-1.csv")
df.head(10)
Out[2]:
1.Data pre-processing¶
In [3]:
df.shape
Out[3]:
In [4]:
df.info()
In [5]:
# All columns have numerical values
# Class would be the target variable. Should be removed when PCA is done
In [6]:
df.isna().sum()
Out[6]:
There are various ways to handle missing values. Drop the rows, replace missing values with median values etc. from above we can say that we have NAN or 0 in the column. We could drop those rows - which might not be a good idea under all situations. Here, we will replace them with their median values.
In [7]:
df['skewness_about'].unique()
Out[7]:
In [8]:
df = df.replace('0', np.nan)
In [9]:
# replace the missing values with median value.
# Note, we do not need to specify the column names below
# every column's missing value is replaced with that column's median respectively
df=df.fillna(df.median())
In [10]:
df.isna().sum()
Out[10]:
In [11]:
df.describe().transpose()
Out[11]:
Observations: Compactness and circularity has mean and median values almost similar , it signifies that it is normally distribited and has no skewness/outlier scatter_ratio feature seems to be having some kind of skewness and outlier
In [12]:
# Check for duplicate data
dups = df.duplicated()
print('Number of duplicate rows = %d' % (dups.sum()))
In [13]:
# Label Encoding
le=preprocessing.LabelEncoder()
df['class']=le.fit_transform(df['class'])
print(df.shape)
df.head()
Out[13]:
2.Understanding the attributes - Find relationship between different attributes (Independent variables) and choose carefully which all attributes have to be a part of the analysis and why¶
A bivariate analysis among the different independent variables can be done using scatter matrix plot. Seaborn libs create a dashboard reflecting useful information about the dimensions
In [14]:
# Check for correlation of variable
df.corr(method='pearson')
Out[14]:
In [15]:
fig=plt.figure(figsize=(15,12))
sns.heatmap(df.corr(),annot=True)#correlation function
Out[15]:
Observations:¶
1.'circularity' is highly correlated with 'max.length_rectangularity' and 'scaled_radius_of_gyration'(0.96 & 0.93)
2.'scatter_ratio' shows high significance with 'pr.axis_rectangularity' ,'max.length_rectangularity', 'scaled_variance','scaled_variance.1','scaled_radius_of_gyration' and 'distance_circulation'
3.'pr.axis_rectangularity' is positively correlated with 'scaled_variance','scaled_variance.1' and 'scaled_radius_of_gyration'
4.'scaled_variance' and 'scaled_variance.1' is also positively correlatedIn [16]:
sns.pairplot(df, diag_kind='kde') # to plot density curve instead of histogram on the diag
Out[16]:
In [17]:
# Remove anyoutliers, standardize variables in pre-processing step
df.boxplot(figsize=(20,3))
Out[17]:
In [18]:
# We could see few outliers here. Possible mode of imputation:
In [19]:
plt.boxplot(df['radius_ratio'])
Out[19]:
In [20]:
z=np.abs(stats.zscore(df))
print(z)
In [21]:
threshold=3
print(np.where(z>3))
In [22]:
print(z[4][4]) #z score higher than 3
In [23]:
df=df[(z<3).all(axis=1)]
In [24]:
df.shape #new shape
Out[24]:
In [25]:
df.boxplot(figsize=(20,3))
Out[25]:
In [26]:
# We could see most of the outliers are now removed.
In [27]:
plt.boxplot(df['radius_ratio'])
Out[27]:
3.Split the data into train and test¶
In [28]:
#Let us break the X and y dataframes into training set and test set. For this we will use
#Sklearn package's data splitting function which is based on random function
from sklearn.model_selection import train_test_split
In [29]:
#now separate the dataframe into dependent and independent variables
#print("shape of new_vehicle_df_independent_attr::",X.shape)
#print("shape of new_vehicle_df_dependent_attr::",y.shape)
X = df.iloc[:,0:18].values
y = df.iloc[:,18].values
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.30, random_state=10)
4.Train a Support vector machine using the train set and get the accuracy on the test set¶
In [30]:
from sklearn import svm
clr = svm.SVC(gamma='scale')
clr.fit(X_train , y_train)
Out[30]:
In [31]:
y_predict = clr.predict(X_test)
In [32]:
# Calculation of accuracy
In [33]:
print("Accuracy on training set: {:.4f}".format(clr.score(X_train, y_train)))
print("Accuracy on test set: {:.4f}".format(clr.score(X_test, y_test)))
In [34]:
model_score = clr.score(X_test, y_test)
print(model_score)
In [35]:
#Store the accuracy results for each model in a dataframe for final comparison
resultsDf = pd.DataFrame({'Method':['svm(raw data)'], 'accuracy': model_score})
resultsDf = resultsDf[['Method', 'accuracy']]
resultsDf
Out[35]:
5.Perform K-fold cross validation and get the cross validation score of the model¶
In [36]:
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
num_folds = 50
seed = 7
kfold = KFold(n_splits=num_folds, random_state=10)
model = LogisticRegression()
results = cross_val_score(model, X, y, cv=kfold)
print(results)
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
In [37]:
kfmodel_score = results.mean()
print(kfmodel_score)
In [38]:
tempResultsDf = pd.DataFrame({'Method':['K-fold cross val(raw data)'], 'accuracy':kfmodel_score })
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'accuracy']]
resultsDf
Out[38]:
6.Use PCA from Scikit learn, extract Principal Components that capture about 95% of the variance in the data¶
In [39]:
# Drop class variables
df_new =df.drop(['class'], axis =1)
df_new.head()
Out[39]:
In [40]:
# Scaling The Independent Data Set
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
df_scaled = sc.fit_transform(df_new)
In [41]:
df_scaled
Out[41]:
In [42]:
covMatrix = np.cov(df_scaled,rowvar=False)
print(covMatrix)
In [43]:
covMatrix.shape
Out[43]:
In [44]:
# Step 2- Get eigen values and eigen vector
eig_vals, eig_vecs = np.linalg.eig(covMatrix)
print('Eigen Vectors \n%s', eig_vecs)
print('\n Eigen Values \n%s', eig_vals)
In [45]:
# Step 2- Get eigen values and eigen vector
eig_vals, eig_vecs = np.linalg.eig(covMatrix)
print('Eigen Vectors \n%s', eig_vecs)
print('\n Eigen Values \n%s', eig_vals) #3.Find variance and cumulative variance by each eigen vector
tot = sum(eig_vals)
var_exp = [( i /tot ) * 100 for i in sorted(eig_vals, reverse=True)]
cum_var_exp = np.cumsum(var_exp)
print("Cumulative Variance Explained", cum_var_exp)
In [46]:
# Make a set of (eigenvalue, eigenvector) pairs:
eig_pairs = [(eig_vals[index], eig_vecs[:,index]) for index in range(len(eig_vals))]
# Sort the (eigenvalue, eigenvector) pairs from highest to lowest with respect to eigenvalue
eig_pairs.sort()
eig_pairs.reverse()
print(eig_pairs)
# Extract the descending ordered eigenvalues and eigenvectors
eigvalues_sorted = [eig_pairs[index][0] for index in range(len(eig_vals))]
eigvectors_sorted = [eig_pairs[index][1] for index in range(len(eig_vals))]
# Let's confirm our sorting worked, print out eigenvalues
print('Eigenvalues in descending order: \n%s' %eigvalues_sorted)
In [47]:
# Ploting
plt.figure(figsize=(10 , 5))
plt.bar(range(1, eig_vals.size + 1), var_exp, alpha = 0.5, align = 'center', label = 'Individual explained variance')
plt.step(range(1, eig_vals.size + 1), cum_var_exp, where='mid', label = 'Cumulative explained variance')
plt.ylabel('Explained Variance Ratio')
plt.xlabel('Principal Components')
plt.legend(loc = 'best')
plt.tight_layout()
plt.show()
Observations: From above we plot we can clealry observer that 8 dimension are able to explain 95 %variance of data.so we will use first 8 principal components going forward and calulate the reduced dimensions
In [48]:
pca = PCA(n_components=8)
pca.fit(df_scaled)
Out[48]:
In [49]:
df_pca=pca.transform(df_scaled)
In [50]:
pca.components_
Out[50]:
In [51]:
df_scaled.shape
Out[51]:
In [52]:
df_pca.shape
Out[52]:
7. Repeat steps 3,4 and 5 but this time, use Principal Components instead of the original data. And the accuracy score should be on the same rows of test data that were used earlier. (hint: set the same random state)¶
In [53]:
# Fitting SVM ON PCA Data
# Split the data into train and test
X1_train, X1_test, y1_train, y1_test = train_test_split(df_pca,y,test_size=0.30, random_state=10)
In [54]:
clr1 = svm.SVC(gamma='scale')
clr1.fit(X1_train , y1_train)
Out[54]:
In [55]:
x_pca_predict = clr1.predict(X1_test)
In [56]:
print("Accuracy on training set: {:.4f}".format(clr1.score(X1_train, y1_train)))
print("Accuracy on test set: {:.4f}".format(clr1.score(X1_test, y1_test)))
In [57]:
model_score1 = clr1.score(X1_test, y1_test)
print(model_score1)
In [58]:
tempResultsDf = pd.DataFrame({'Method':['svm(pca data)'], 'accuracy': [model_score1]})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'accuracy']]
resultsDf
Out[58]:
In [59]:
# K-fold cross validation on PCA data
num_folds = 50
seed = 7
kfold1 = KFold(n_splits=num_folds, random_state=seed)
model1 = LogisticRegression()
results = cross_val_score(model1, df_pca, y, cv=kfold1)
print(results)
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
In [60]:
kfmodel_score1 = results.mean()
print(kfmodel_score1)
8. Compare the accuracy scores and cross validation scores of Support vector machines – one trained using raw data and the other using Principal Components, and mention your findings (5 points)¶
In [61]:
tempResultsDf = pd.DataFrame({'Method':['K-fold cross val(pca data)'], 'accuracy':kfmodel_score1})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'accuracy']]
resultsDf
Out[61]:
Observations:
- On testing data set we saw that our support vector classifier without performing PCA has an accuracy score of 66 % But when we applied the SVC model on PCA componenets(reduced dimensions) our model scored 91 %.
- When we appliy k.fold crss validation on raw data we get 95% accuracy. 3..Considering that original dataframe had 18 dimensions and After PCA dimension reduced to 8, our model has fared well in terms of accuracy score. 4.By reducing dimension to 8 we get accuracy 84% with std (7.3%)